TWISTED BURNSIDE-FROBENIUS THEORY 
FOR DISCRETE GROUPS 



ALEXANDER FEL'SHTYN AND EVGENIJ TROITSKY 

Abstract. For a wide class of groups including polycyclic and finitely generated poly- 
nomial growth groups it is proved that the Reidemeister number of an automorphism 
is equal to the number of finite-dimensional fixed points of the induced map (j) on the 
unitary dual, if one of these numbers is finite. This theorem is a natural generalization 
of the classical Burnside-Frobenius theorem to infinite groups. This theorem also has 
important consequences in topological dynamics and in some sense is a reply to a remark 
of J. -P. Serre. The main technical results proved in the paper yield a tool for a further 
progress. 
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1. Introduction 

Definition 1.1. Let G be a countable discrete group and : G ^ G an endomorphism. 
Two elements x,x' & G are said to be (p-conjugate or twisted conjugate, if and only if there 
exists g E G with 

x' = gx(f){g~'^). 

We will write {x}^ for the (p- conjugacy or twisted conjugacy class of the element x G G. 
The number of 0-conjugacy classes is called the Reidemeister number of an endomorphism 
and is denoted by R{4>)- If 4> is the identity map then the 0-conjugacy classes are the 
usual conjugacy classes in the group G. 
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If G is a finite group, then the classical Burnside-Frobenius theorem (see, e.g., [40], 
[27, p. 140]) says that the number of classes of irreducible representations is equal to the 
number of conjugacy classes of elements of G. Let G be the unitary dual of G, i.e. the 
set of equivalence classes of unitary irreducible representations of G. 

li (f) : G G is an automorphism, it induces a map (p : G ^ G, (j){p) = po0. Therefore, 
by the Burnside-Frobenius theorem, if is the identity automorphism of any finite group 
G, then we have -R(0) = #Fix(0). 

In [7] it was discovered that this statement remains true for any automorphism of any 
finite group G. Indeed, if we consider an automorphism of a finite group G, then -R(0) is 
equal to the dimension of the space of twisted invariant functions on this group. Hence, by 
Peter- Weyl theorem (which asserts the existence of a two-side equivariant isomorphism 
C*{G) = 0^gg End(-ffp)), -R(0) is identified with the sum of dimensions dp of twisted 

invariant elements of End(i?p), where p runs over G, and the space of representation p 
is denoted by Hp. By the Schur lemma, dp = 1, if p is a fixed point of 0, and is zero 
otherwise. Hence, -R(0) coincides with the number of fixed points of 0. 

Dynamical questions have inspired a series of papers [7, 8, 4, 12, 14, 10], attempting 
to generalize this theorem to the cases of non-identical automorphisms and of non-finite 
groups. In these papers a version of the theorem for almost Abelian groups is proved, and 
some examples and particular cases are considered. 

In the present paper we introduce the property RP (Definition 3.6) for a countable 
discrete group G: the 0-class functions of any automorphism with -R(0) < oo are 
periodic in a natural sense. 

After some preliminary and technical considerations we prove the main results of the 
paper, namely 

(1) RP respects some extensions: Suppose there is an extension H ^ G ^ G/ H, 
where the group if is a characteristic RP-group; G/H is finitely generated FC- 
group (i.e. a group with finite conjugacy classes). Then G is an RP-group (a 
reformulation of Theorem 3.10). 

(2) Classes of RP groups: Polycyclic groups and finitely generated groups of poly- 
nomial growth are RP-groups. Moreover, almost-polycyclic groups are RP too. 
(Theorems 4.2, 4.4, 4.6). The Twisted Burnside-Frobenius theorem is valid for 
them (Theorem 4.5). 

(3) Twisted Burnside-Frobenius theorem for RP-groups: Let G be an RP 

group and its automorphism with -R(0) < oo. Denote by G/ the subset of the 
unitary dual G related to finite-dimensional representations. Denote by 5'/(0) the 
number of fixed points of 0/- on G/. Then -R(0) = 5'/(0). (Theorem 5.2). 

(4) Twisted Burnside-Frobenius theorem for almost polycyclic groups: Let 
G be an almost polycyclic group. Then -R(0) = >S'/(0) if one of these numbers is 
finite (Theorem 7.8). 

In some sense our theory is a reply to a remark of J. -P. Serre [40, (d), p. 34] that for 
compact infinite groups, an analogue of the Burnside-Frobenius theorem is not interesting: 
oo = cxD. It turns out that for infinite discrete groups the situation differs significantly, 
and even in non-twisted situations the number of classes can be finite (for one of the 
first examples see another book of J. -P. Serre [41]). Several examples of groups and 
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automorphisms with finite Reidemeister numbers were obtained and studied in [4, 18, 9, 
14, 10]. 

Using the same argument as in [12] one obtains from the twisted Burnside-Frobenius 
theorem the following dynamical and number-theoretical consequence which, together 
with the twisted Burnside-Frobenius theorem itself, is very important for the realization 
problem of Reidemeister numbers in topological dynamics and the study of the Reide- 
meister zeta-f unction. 

Let fi{d), d G N, be the Mohius function, i.e. 

r 1 if ci = 1, 
= < (—1)*^ if (i is a product of k distinct primes, 
[^0 if (i is not square — free. 

Congruences for Reidemeister numbers: Let (p : G ^ G be an automorphism 
of a countable discrete KP-group G such that all numbers -R(0") are finite. Then one has 
for all n, 

^/i(rf)-i?(0"/'^) = O modn. 

d\n 

These theorems were proved previously in a number of special cases in [7, 8, 12, 14, 10]. 
We would like to emphasize the following important remarks. 

(1) In the original formulation by Fel'shtyn and Hill [7] the conjecture about twisted 
Burnside-Frobenius theorem asserts an equality of -R(0) and the number of fixed 
points of on G. This conjecture was proved in [7, 12] for f.g. type I groups. 

(2) In our paper [14] with A. Vershik, we studied a key example which shows that 
an RP-group can have infinite-dimensional "supplementary" fixed representations. 
Our example was a semi-direct product of the action of Z on Z©Z by a hyperbolic 
automorphism. We consider an automorphism with finite Reidemeister number 
(four to be precise), (f) has at least five fixed points on G, but exactly four fixed 
points on G/. 

This gives a counterexample to the conjecture in its original formulation (in 
which we count all fixed points in G) and leads to the formulation, in which we 
count fixed points only from Gj. This new conjecture is proved in the present 
paper for a wide class of f.g. groups. 

(3) The extra-fixed-point phenomenon arises from bad separation properties of G for 
a general discrete group G. A deeper study leads to the following general theorem. 

Weak Twisted Burnside theorem [A^:Let R^.{(t)) be the number of Rei- 
demeister classes related to twisted invariant functions on G from the Fourier- 
Stieltjes algebra B{G). Let S^,{(f)) be the number of generalized fixed points of (p 
on the Glimm spectrum ofG, i. e., on the complete regularization ofG. If one of 
R^:{(f)) and S'*(0) is finite, then R^,{(j)) = S'*(0). 

The proof is based on a non- commutative version of the well-known Riesz(- 
Markov-Kakutani) theorem, which identifies the space of linear functionals on the 
algebra A = G{X) with the space of regular measures on X. To prove the Weak 
Twisted Burnside theorem we first obtain a generalization of the Riesz theorem to 
the case of a non-commutative C*-algebra A using the Dauns-Hofmann sectional 
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representation theorem (in the same paper [44]). The corresponding measures on 
the Ghmm spectrum are functional-valued. In extreme situations this theorem 
is tautological. But for many cases of group C*-algebras of discrete groups one 
obtains some new method for counting twisted conjugacy classes. This leads to 
an approach alternative to the one we present here. 
(4) The main Theorem 3.10 allows us to verify the periodicity of </)-class functions in a 
number of cases which are not in the classes described in Section 4. Nevertheless 
for pathological groups from Section 6 even the modified conjecture is not true. 
Keeping in mind that for Gromov hyperbolic groups R{4>) is always infinite (as 
well as for Baumslag-Solitar groups and some generalizations, cf. [5, 30, 6, 29, 43]) 
while in the "opposite" case the twisted Burnside theorem is proved we hope that 
various use of Theorem 3.10 can lead to a complete resolution of the problem, if 
the groups from Section 6 will be handled. 

The interest in twisted conjugacy relations has its origins, in particular, in the Nielsen- 
Reidemeister fixed point theory (see, e.g. [25, 4]), in Selberg theory (see, eg. [42, 1]), and 
Algebraic Geometry (see, e.g. [21]). 

Concerning some topological applications of our main results, they are already described 
in [12]. The congruences give some necessary conditions for the realization problem for 
Reidemeister numbers in topological dynamics. The relation to Selberg theory will be 
presented in a forthcoming paper. 
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2. Preliminary Considerations 

The following fact will be useful. 

Theorem 2.1 ([37, Theorem 1.41]). IfG is a finitely generated group and H is a subgroup 
with finite index in G, then H is finitely generated. 

Lemma 2.2. Let G he finitely generated, and H' G G its subgroup of finite index. Then 
there is a characteristic subgroup H G G of finite index, H G H' . 

Proof. Since G is finitely generated, there is only finitely many subgroups of the same 
index as H' (see [23], [28, § 38]). Let H be their intersection. Then H is characteristic, 
in particular normal, and of finite index. □ 

Lemma 2.3. Let G be abelian. The twisted conjugacy class H of e is a subgroup. The 
other classes are cosets gH. 

Proof. The first statement follows from the equalities 

hcj>{h-')gcj>{g-') = gh(i>iighr\ mh-')r' = mh~' = h-'m, 
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where h E H. For the second statement, suppose a ^ b, i.e. b = ha(j)[h~^). Then 

gb = gha(l){h~^) = h{ga)(p{h~^), gb ~ ga. 

□ 

Let us denote by Tg : G ^ G the automorphism Tg(g) = gg g~^ for g E G. Its restriction 
to a normal subgroup we will denote by Tg as well. 

Lemma 2.4. {g}^k = {g k}r^^-^o4>- 

Proof. Let g' = f g 4>{f~^) be ^-conjugate to g. Then 

9'k = fg k = fgkk-' A; = / ((7 fc) (r,-i o 0) (/"i). 

Conversely, if g' is (r^-i o 0)-conjugate to g, then 

^7' = fg{T,-. o 0)(/-i)A;-i = / A;"! <j>{r'). 
Hence a shift maps </)-conjugacy classes onto classes related to another automorphism. □ 
Corollary 2.5. R{4>) = R{Tg o 0). 

Theorem 2.6 (see [25]). Let A be a finitely generated Abelian group, ip : A A its 
automorphism. Then R{ip) = # Coker (■?/; — Id) , i.e. to the index of subgroup generated by 
elements of the form x'^iplx). 

Proof. By Lemma 2.3, R{ip) is equal to the index of the subgroup H = {e}^. This group 
consists by definition of elements of the form x~^ip{x). □ 

The following construction relates 0-conjugacy classes and some conjugacy classes of 
another group. It was obtained in topological context by Boju Jiang and Laixiang Sun 
in [26]. Consider the action of Z on G, i.e. a homomorphism Z —>■ Aut(G'), n i— > (p^. Let 
r be the corresponding semi-direct product F = G x Z: 

(1) T:=<G,t\tgt-' = (P{g)> 

in terms of generators and relations, where t is a generator of Z. The group G is a normal 
subgroup of F. As a set, F has the form 

(2) F = UnezG ■ r, 
where G ■ t" is the coset by G containing t"^. 

Remark 2.7. Any usual conjugacy class of F is contained in some G ■ f^. Indeed, 
gg'f'g'^ = gg'^''{g-^)t'' and tg'f't-^ = <p{g')t''. 

Lemma 2.8. Two elements x, y ofG are (p-conjugate if and only if xt andyt are conjugate 
in the usual sense in F. Therefore g ^ g ■ t is a bijection from the set of (p- conjugacy 
classes of G onto the set of conjugacy classes ofT contained in G ■ t. 

Proof. If X and y are 0-conjugate then there is a G G such that gx = y(p{g)- This 
implies gx = ytgt~^ and therefore g{xt) = {yt)g so xt and yt are conjugate in the usual 
sense in F. Conversely, suppose xt and yt are conjugate in F. Then there is a gt"' G F 
with gf^xt = ytgt"'. From the relation txt~^ = (p{x) we obtain g(p^{x)t'^^^ = y(p{g)t'^^^ 
and therefore g(p"'{x) = y(p{g). Hence, y and 0"(x) are 0-conjugate. Thus, y and x are 
^-conjugate, because x and (p{x) are always 0-conjugate: 0(x) = x~^x(p{x). □ 
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3. Extensions and Reidemeister Classes 
Consider a group extension respecting homomorphism 0: 

(3) H — G — G/H 




where if is a normal subgroup of G. The argument below, especially concerning the role 
of fixed points, has a partial intersection with [18, 16, 17]. 

First, notice that the Reidemeister classes of in G are mapped epimorphically onto 
classes of in G/H. Indeed, 

(4) p{g)p{9Mp(9~^)) = p(gg<P{T^)- 

Suppose, -R(0) < oo. Then the previous remark implies < oo. Consider a class 

D = {/i}rg(/)', where Tg{h) = ghg~^, g E G, h E H . The corresponding equivalence relation 
is 

(5) hr~^hhg(f)'(h-^)g-\ 

Since H is normal, the automorphism Tg : H H is well defined. We will denote by D 
the image iD as well. By (5) the shift Dg is a subset of Hg, characterized by 

(6) hgr^hihg)<l)'(h"'). 

Hence it is a subset of {hg}^ fl Hg. The partition Hg = U{{h}T-^ipi)g is a subpartition of 
Hg = U{Hgn{hg}^). 

We need the following lemmas. 

Lemma 3.1. Suppose the extension (3) satisfies the following conditions: 

(1) #Fix0 = A; < oo, 

(2) R{(f)) < oo. 

Then 

(7) R{(P') <k-{R{(P)-R(^) + l). 

If G/H is ahelian, let gi he some elements withp{gi) being representatives of all different 
(f)-conjugacy classes, i = 1, . . . , R{(f)). Then 

(8) 5^i?(r,^0')<fc-W)- 

i=l 

Proof Consider classes z E G, i.e. the classes of relation z ~ hz(f)'{h~^), h E H. 

The group G acts on them hj z ^ gz(j){g~^). Indeed, 

g[hh<j>{h~'Mg~') = {ghg~'){gh<j>{g-')mg)<P{h~')<l>{g^'^^ 

= ighg-')igh<Pig-'mghg~') E {gh<P{g-')}^,, 

because H is normal and ghg~^ E H . Due to invertibility, this action of G transposes 
classes {z}^i inside one class {g},),- Let d denote the number of classes {/i}^' inside 
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{h}(f, n H. Then d does not exceed the number of G G such that p{g)4>{p{g ^)) = e. 
Since two elements g and gh in one iJ-coset induce the same permutation of classes {h}^r, 
d does not exceed the number of z E G/H such that z(f){z~^) = e, i.e. d < k. This implies 

(7) . 

Now we discuss ^-classes over 0-classes other than {e}-^ for an abelian G/H. An estima- 
tion analogous to the one above leads to the number of z E G/H such that zzo4>{z~^) = zq 
for some fixed zq- But for an Abelian G/H they form the same group Fix(0). This, to- 
gether with the description (6) of shifts of D at the beginning of this Section, implies 

(8) . □ 

Lemma 3.2. Suppose, in the extension (3) the group H is abelian. Then ^ Fix(0) < 
#Fix(0')-#Fix(0). 

Proof. Let s : G/H ^ G be a section of p. If s{z)h is a fixed point of then 

(9) {s{z))-'<P{s{z)) = h<P\h-'). 

Hence, z e Fix(0) and left hand side takes k := #Fix(</)) values hi,...,hk. Let us 
estimate the number of s{z)h for a fixed z such that {s{z))~^(j){s{z)) = hi. These h have 
to satisfy (9). Since H is abelian, if one has 

hi = h(f)'{h~^) =h(f)'(h~^), 

then h~^h G Fix(</)') and we are done. □ 

Theorem 3.3. Let A be a finitely generated Abelian group, ip : A A its automorphism 
with R{ip) < oo. Then ^Fix{ip) < oo. 
Moreover, R{'4)) > ^Fix{tlj). 

Proof. Let T be the torsion subgroup. It is finite and characteristic. We obtain the 
extension T —>■ A ^ A/T respecting </>. Since A/T = Z^ we have Fix(V^ : A/T ^ 
A/T) = e, by [25], [4, Sect. 2.3.1]. Hence, by Lemma 3.2, #Fix(?^) < #Fix(V^'), where 
ip' : T ^ T. For any finite abelian group T one clearly has # Fix(?/^') = R{tp') by Theorem 
2.6 (cf. [4, p. 7]). Finally, < R{ip) by (8). □ 

Recall the following definitions of a class of groups. 

Definition 3.4. A group with finite conjugacy classes is called FC-group. 

In an FC-group, the elements of finite order form a characteristic subgroup with locally 
infinite abelian factor group; a finitely generated FC-group contains in its center a free 
abelian group of finite index in the whole group [32]. 

Lemma 3.5. An automorphism (j) of a finitely generated FC-group G with R{(f)) < oo has 
a finite number of fixed points. 

The same is true for o (p. Hence, the number of g E G, such that for some x E G, 

gx(j){g~^) = X, 

remains finite. 
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Proof. Let A be the center of G. As was indicated above, A has finite index in G and hence, 
by Theorem 2.1, is f.g. Since A is characteristic, one has an extension A G ^ G/A 
respecting 0. Hence, < ■ \G/A\ by Lemma 3.1 and #Fix(0') < R{(p) ■ \G/A\ 

by Theorem 3.3. Then #Fix(0) < R{(j)) ■ IG/Ap by Lemma 3.2. □ 

Definition 3.6. We say that a group G has the property RP if for any automorphism 
(p with R{(f)) < oo the characteristic functions / of Reidemeister classes (hence all 
^-central functions) are periodic in the following sense. 

There exists a finite group K, its automorphism (px, and epimorphism F : G ^ K 
such that 

(1) The diagram 




commutes. 

(2) / = F*fK, where fx is a characteristic function of a subset of K. 
If this property holds for a concrete automorphism (p, we will denote this property by 
RP(0). 

Remark 3.7. By (2) there is only one class {g}(f, which maps onto {F{g)}^^. Hence, F 
induces a bijection of Reidemeister classes. 

Lemma 3.8. Suppose, G is f.g. and R{(j)) < oo. Then characteristic functions of 0- 
conjugacy classes are periodic {i.e. G has RP(0) ) if and only if their left shifts generate 
a finite dimensional space. 

Proof. By the supposition of finite dimension it follows that the stabilizer of each (p- 
conjugacy class has finite index. Hence, the common stabilizer of all (/)-conjugacy classes 
under left shifts is an intersection of finitely many subgroups, each of finite index. Hence, 
its index is finite. By Lemma 2.2 there is some smaller subgroup Gs oi finite index which 
is normal and 0-invariant. Then one can take K = G/Gs- Indeed, it is sufficient to verify 
that the projection F is one to one on classes. In other words, that each coset of Gg enters 
only one 0-conjugacy class, or any two elements of coset are 0-conjugated. Consider g 
and kg, g & G, h & Gg- Since h by definition preserves classes, kg = xgcp{x~^) for some 
X e G, as desired. 

Conversely, if G has RP(0), the class {g}^ is a full pre-image F~^{S) of some class 
S <Z K. Then its left shift can be described as 

g'{g}^ = g'F~\S) = {g'g^\g^eF~\S)} = {g\{gr'g^F-\S)} 

= {glFiigT'g) eS} = {g\Fig) e Fig')iS)} = F-\Fig'){S)). 

Since K is finite, the number of these sets is finite. □ 

Remark 3.9. 1) In this situation in accordance with Lemma 2.2 the subgroup Gs is 
characteristic, i.e. invariant under any automorphism. 

2) Also, the group G/Gs will serve as K (i.e. give rise to a bijection on sets of Reide- 
meister classes) not only for (p but for Tg o cp for any g & G because they have the same 
collection of left shifts of Reidemeister classes by Lemma 2.4. 
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Theorem 3.10. Suppose, the extension (3) satisfies the following conditions: 

(1) H has RP; 

(2) G/H IS FC f.g. 

Then G has RPf^j. 

Proof. We have -R(0) < oo, hence # Fix(0) < oo by Lemma 3.5 as weU as # Fix(r^</)) < oo 
for any z G G/H. Then by Lemma 3.1 R{Tg(f)') < oo for any g E G. Let gi,...,gs, 
s = R{4>), be elements of G which are mapped by p to different Reidemeister classes of 
(j). Now we can apply the supposition that H has RP and find a characteristic subgroup 
Hk '■= KerF G H oi finite index such that F : H ^ K gives rise to a bijection for 
Reidemeister classes of each of the automorphisms r^. o 0', z = 1, . . . , s. Moreover, Hk 
is contained in the stabilizer (under left shifts) of each twisted conjugacy class of each of 
these automorphisms. In particular, it is normal in G. Hence, we can take a quotient by 
Hk of the extension (3): 

H"— 

F 

H/Hk^ 

Gi — ^r. 

The quotient map Fi : G — > G/Hk takes {g}ct, to {g}^ and this class is a unique class 
with this property (we conserve the notations e,g,(f) for the quotient objects). Indeed, 
suppose two classes are mapped onto one. This means that there are two elements g and 
ghx of these different classes, hx G Hk- One can choose g E G such that gg(t>(g~^) = gih 
for some i = 1, . . . , s, h E H . Then 

9{9hK)<P{9~^) = gg~^gM{g)hK(l){g^^) = gikhx, hx e Hk- 

Hence, gih and gihhx belong to the same (but different) classes as g and ghx- Moreover, 
they can not be 0-conjugate by elements of H . Hence (cf. (6)), the elements h and hh^ 
are not (r^. o (/)')-conjugate in H. But Hk 9 hx is a subgroup of the intersection of 
stabilizers (see above). We arrive to a contradiction. 

By Lemma 3.8 (applied to Gi and concrete automorphism 0) for the purpose of finding 
a map F2 : Gi ^ Ki with properties (1) and (2) of the Definition 3.6 it is sufficient to 
verify that shifts of the characteristic function of {/i},/, C Gi form a finite dimensional 
space, i.e. the shifts of {h}^ C Gi form a finite collection of subsets of Gi. After that one 
can take the composition 

to complete the proof of theorem. 

We were able to apply Lemma 3.8 above, because the group Gi is finitely generated. 
For example, we can take as generators all elements of K and some pre-images s{zi) G Gi 
under p of a finite system of generators Zi for F. Indeed, for any a; G Gi we can find some 
product of Zi to be equal to p{x). Then the same product of s{zi) differs from x by an 
element of K. 
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Let us prove that the mentioned space of shifts is finite-dimensionaL By Lemma 2.4 
these shifts of {/i}^ C Gi form a subcollection of 

{x}ryo<t„ x,yeGi. 

Hence, by Corollary 2.5 it is sufficient to verify that the number of different automorphisms 
Ty : Gi ^ Gi is finite. 

Let Xi, . . . ,Xn be some generators of Gi. Then the number of different Ty does not 
exceed 

n n 

n \yeG-,}<l[\K\- #{r,(p(x,)) U G L}, 

i=i i=i 
where the last numbers are finite by the definition of FC for F. □ 

4. PoLYCYCLic Groups and Groups of Polynomial Growth 

Now we will prove using Theorem 3.10 that some classes of groups are RP groups. 
As one could expect, these classes are only a small part of possible applications of this 
theorem. 

Let G' = [G, G] be the commutator subgroup or derived group of G, i.e. the subgroup 
generated by commutators. G' is invariant under any homomorphism, in particular it is 
normal. It is the smallest normal subgroup of G with an abelian factor group. Denoting 
G^o) := G, G(^) := G', G(") := (G^^-^))', n > 2, one obtains denved series of G: 

(10) G = G(°) D G' D G(2) D . . . D G(") D ... 

If G(") = e for some value n, i.e. the series (10) stabilizes by trivial group, then the group 
G is solvable. 

Definition 4.1. A solvable group is a polycyclic group, if it has a derived series with 
cyclic factors. 

Theorem 4.2. Any polycyclic group is RP. 

Proof. By Lemma 2.3 any commutative group is RP. Any extension with H being the 
commutator subgroup G' of G respects any automorphism (p of G, because G' is evidently 
characteristic. The factor group is abelian, in particular FC. 

Any polycyclic group is a result of a finite number of extensions with finitely gener- 
ated (cyclic) factor groups, starting from Abelian group. Thus we can apply inductively 
Theorem 3.10 to complete the proof. □ 

Theorem 4.3. Any finitely generated nilpotent group is RP. 

Proof. These groups are supersolvable, hence, polycyclic [38, 5.4.6, 5.4.12]. □ 

Theorem 4.4. Any finitely generated group of polynomial growth is RP. 

Proof. By [20] a finitely generated group of polynomial growth is just a finite extension 
of an f.g. nilpotent group H. The subgroup H can be supposed to be characteristic, 
i.e. (p^H) = H for any automorphism cf) : G ^ G. Indeed, let if' C G be a nilpotent 
subgroup of index j. Let H be the subgroup from Lemma 2.2. By Theorem 2.1 it is 
finitely generated. Also, it is nilpotent as a subgroup of nilpotent group (see [28, § 26]). 

Since a finite group is a specific case of FC group and an f.g. nilpotent group has RP 
by Theorem 4.3, we can apply Theorem 3.10 to complete the proof. □ 
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Theorem 4.5. The Reidemeister number of any automorphism (p of an f.g. group of 
polynomial growth or polycyclic group with R{(j)) < oo is equal to the number of finite- 
dimensional fixed points of on the unitary dual of this group. 

This theorem wiU be proved in a more general form below (Theorem 5.2). 

Theorem 4.6. Any almost polycyclic group, i.e. an extension of a polycyclic group with 
a finite factor group, is RP. 

Proof. The proof repeats almost literally the proof of Theorem 4.4. One has to use the 
fact that a subgroup of a polycyclic group is polycyclic [38, p. 147]. □ 

These theorems will be proved below in a more strong form (Theorem 7.8). 

5. The Twisted Burnside-Frobenius Theorem for RP Groups 

Definition 5.1. Denote by Gj the subset of the unitary dual G related to finite-dimen- 
sional representations. 

Theorem 5.2 (Twisted Burnside-Frobenius Theorem for RP-groups). Let G be an RP 

group and (p its automorphism with R{(f)) < oo. Denote by 5/(0) the number of fixed 

points of (f)f onGf. Then 

R{<P) = Sf{<P). 

Proof. The coefficients of finite-dimensional non-equivalent irreducible representations of 
G are linear independent by Frobenius-Schur theorem (see [2, (27.13)]). Moreover, the 
coefficients of non-equivalent unitary finite-dimensional irreducible representations are 
orthogonal to each other as functions on the universal compact group associated with the 
initial group [3, 16.1.3] by the Peter- Weyl theorem. Hence, their linear combinations are 
orthogonal to each other as well. 

It is sufficient to verify the following three statements: 

1) If -R(0) < oo, then each 0-class function is a finite linear combination of twisted- 
invariant functionals being coefficients of points of Fix0j. 

2) If p G Fix 0/, there exists one and only one (up to scaling) twisted invariant functional 
on p(C*(G)) (this is a finite full matrix algebra). 

3) For different p the corresponding 0-class functions are linearly independent. This 
follows from the remark at the beginning of the proof. 

Let us remark that the property RP implies in particular that </)-central functions (for (f) 
with -R(0) < oo) are functionals on G*{G), not only L}{G), i.e. are in the Fourier-Stieltijes 
algebra B{G). 

The statement 1) follows from the RP property. Indeed, this </)-class function / is a 
linear combination of functionals coming from some finite collection {pi} of elements of 
Gf (these representations pi, . . . , ps are in fact representations of the form tTj oF, where tTj 
are irreducible representations of the finite group K and F : G ^ K, a.s in the definition 
of RP). So, 

s 

/ = 5^/.°P., pr-G ^EndiV^, /, : End(V,) ^ C, p,^p^,{t^j). 
1=1 
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For any g,g G G one has 

s s 

Y.mp^(9)) = fig) = = Y.f^^p^^9m9-')))- 

1=1 i=l 

By the observation at the beginning of the proof concerning hnear independence, 

Mpiig)) = fi{pi{99<P{9~^)))- 2 = 1, . . . , s, 

i.e. fi are twisted-invariant. For any p G G/, p : G ^ End(V^), any functional ui : 
End(y) — >■ C has the form a i-^ Tr(6a) for some fixed b G End(y). Twisted invariance 
imphes twisted invariance of b (evident details can be found in [12, Sect. 3]). Hence, 
b is intertwining between p and p o and p G Fix(0j). The uniqueness of intertwining 
operator (up to scaling) implies 2). □ 

6. Counterexamples 

Now let us consider some counterexamples to this statement for pathological (monster) 
discrete groups. Suppose, an infinite discrete group G has a finite number of conjugacy 
classes. Such examples can be found in [41] (HNN-group), [33, p. 471] (Ivanov group), 
and [34] (Osin group). Then evidently, the characteristic function of unity element is not 
almost-periodic and the argument above is not valid. Moreover, let us show, that these 
groups produce counterexamples to the above theorem. 

Example 6.1. For the Osin group the Reidemeister number i?(Id) = 2, while it has only 
one finite-dimensional irreducible representation (trivial 1-dimensional). Indeed, the Osin 
group is an infinite, finitely generated group G with exactly two conjugacy classes. All 
nontrivial elements of this group G are conjugate. So, the group G is simple, i.e. G 
has no nontrivial normal subgroup. This implies that group G is not residually finite 
(by definition of residually finite group). Hence, it is not linear (by Mal'cev theorem 
[31], [38, 15.1.6]) and has no finite-dimensional irreducible unitary representations with 
trivial kernel. Hence, by simplicity of G, it has no finite-dimensional irreducible unitary 
representation with nontrivial kernel, except of the trivial one. 

Let us remark that the Osin group is non-amenable, contains the free group in two 
generators F2, and has exponential growth. 

Example 6.2. For large enough prime numbers p, the first examples of finitely generated 
infinite periodic groups with exactly p conjugacy classes were constructed by Ivanov as 
limits of hyperbolic groups (although hyperbolicity was not used explicitly) (see [33, 
Theorem 41.2]). The Ivanov group G is an infinite periodic 2-generator group, in contrast 
to the Osin group, which is torsion free. The Ivanov group G is also a simple group. 
The proof (kindly explained to us by M. Sapir) is the following. Denote by a and b the 
generators of G described in [33, Theorem 41.2]. In the proof of Theorem 41.2 on [33] it 
was shown that each of elements of G is conjugate in G to a power of generator a of order 
s. Let us consider any normal subgroup of G. Suppose 7 G A^. Then 7 = ga'^g~^ for 
some g & G and some s. Hence, = g'^^ig G A^ and from periodicity of a, it follows that 
also a G A^ as well as a'^ G A^ for any /c, because p is prime. Then any element h of G also 
belongs to A^ being of the form h = ha^{h)~^, for some /c, i.e., N = G. Thus, the group 
G is simple. The discussion can be completed in the same way as in the case of the Osin 
group. 
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Example 6.3. In paper [24], Theorem III and its corollary, G. Higman, B. H. Neumann, 
and H. Neumann proved that any locally infinite countable group G can be embedded 
into a countable group G* in which all elements except the unit element are conjugate to 
each other (see also [41]). The discussion above related Osin group remains valid for G* 
groups. 

Let us remark that almost polycyclic groups are residually finite (see e.g. [38, 5.4.17]) 
while the groups from these counterexamples are not residually finite, as it was shown by 
definition. Hence, we would like to complete this section with the following question. 
Question. Suppose G is a residually finite group and is its endomorphism with finite 
R{<p). Does R{<p) equal 5/(0)? 

7. Twisted Conjugacy Separateness 

In fact, the notion of RP group is closely related to a generalization of the following 
well-known notion. 

Definition 7.1. A group G is conjugacy separable if any pair g, h of non-conjugate 
elements of G are non-conjugate in some finite quotient of G. 

It was proved that almost polycyclic groups are conjugacy separable ([35, 15], see also 
[39, Ch. 4]). Also, residually finite recursively presented Burnside p-groups constructed 
by R. I. Grigorchuk [19] and by N. Gupta and S. Sidki [22] are shown to be conjugacy 
separable when p is an odd prime in [45]. 

We can introduce the following notion, which coincides with the previous definition in 
the case = Id. 

Definition 7.2. A group G is (^-conjugacy separable with respect to an automorphism 
: G — s> G if any pair g, h of non-0-conjugate elements of G are non-0-conjugate in some 
finite quotient of G respecting 0. 

One gets immediately the following statement. 

Theorem 7.3. Suppose, R{(f)) < oo. Then G is (p-conjugacy separable if and only if G is 
RP(0). 

Proof. Indeed, let Fij : G Kij distinguish ith and jth 0-conjugacy classes, where Kij 
are finite groups, i,j = l,...,/2(0). Let F : G ^ ®ijKij, F{g) = ^ijFij{g), be the 
diagonal mapping and K its image. Then F : G K gives RP(0). 

The opposite implication is evident. □ 

Theorem 7.4. Let F : T K be a morphism onto a finite group K which separates two 
conjugacy classes ofV in G ■ t. Then the restriction Fq := F\c : G lm{F\G) separates 
the corresponding {by the bijection from Lemma 2.8) (p-conjugacy classes in G. 

Proof. First let us remark that Ker(FG') is 0-invariant. Indeed, suppose Fclg) = F{g) = e. 
Then 

FciM) = F{<P{g)) = F{tgt-') = F{t)F{tr' = e 
(the kernel of F is a normal subgroup). 

Let gt and gt be some representatives of the mentioned conjugacy classes. Then 

F{{ht'')gt{ht''y^) ^ F(gt), V/i G G, n G Z, 
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F{hegt) 7^ F(gthe), 
F{hr{g)) ^ Fmh)), 



V/i G G, 72 G Z, 

\fheG,neZ, 
yheG,neZ, 



in particular, F{hg^{h'^)) F(g) Wh G G. 



□ 



Theorem 7.5. Let some class of conjugacy separable groups be closed under taking semidi- 
rect products by Z. Then this class consists o/RP groups. 



As an application we obtain another proof of the main theorem for almost polycyclic 
groups. 

Theorem 7.6. Any almost polycyclic group is an RP group. 

Proof. The class of almost polycyclic groups is closed under taking semidirect products 
by Z. Indeed, let G be an almost polycyclic group. Then there exists a characteristic 
(polycyclic) subgroup P of finite index in G. Hence, P x Z is a polycyclic normal group 
of G X Z of the same finite index. 

Almost polycyclic groups are conjugacy separable ([35, 15], see also [39, Ch. 4]). It 
remains to apply Theorem 7.5. □ 

Remark 7.7. It is clear that this approach (i.e. the using of Theorem 7.5) can be extended 
to a number of situations (cf. [36]). The results of the present paper are generalized to 
the case of two endomorphisms and their coincidences in [11]. The method of proof there 
is that of Sections 3-5. The method of Section 7 does not work in this case. 

In fact, in [11] it is obtained in particular a generalization of the main result to the 
case of endomorphism. For the method of Sections 3-5 the condition of finiteness of 
Reidemeister number is very important. On the other hand for the method of Section 7 
the property of (j) to be an automorphism is very important. Moreover, we can obtain the 
following final form of the twisted Burnside-Frobenius theorem for almost polycyclic and 
some other classes of groups. 

Theorem 7.8 (Twisted Burnside-Frobenius theorem for 0-conjugacy separable groups). 
Let G be an almost polycyclic group {or, more generally, any group of a class under the 
hypothesis of Theorem 7.5, or even more generally, a (p-conjugacy separable group). Then 
R{4>) = Sf{(f)) if one of these numbers is finite. 

Proof. It remains to prove that R{(f)) < oo if Sf{(f)) < oo. By the definition of a 0- 
conjugacy separable group the Reidemeister classes of can be separated by maps to fi- 
nite groups. Hence, taking representations of these finite groups and applying the twisted 
Burnside-Frobenius theorem to these groups we obtain that for any pair of Reidemeister 
classes there exists a function being a coefficient of a finite-dimensional unitary representa- 
tion, which distinguish these classes. Hence, if R{(f)) = oo, then there are infinitely many 
linearly independent twisted invariant functions being coefficients of finite dimensional 
representations. But there are as many such functionals, as Sf{(f)). □ 

Remark 7.9. As it is clear from the consideration in Section 6, the Osin group and its 
predecessors are not conjugacy separable. In this relation it is natural to reformulate the 
question from Section 6 in the following form. 



Proof. This follows immediately from Theorem 7.4 and Theorem 7.3. 



□ 
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Question. Suppose G is a residually finite group and </> is its endomorphism with finite 
Does this imply that G is 0-conjugacy separable ? 
Some other aspects of the twisted analog of Burnside-Frobenius theory are studied in 
[13]. 
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